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Abstract 

Authentic and constructive learning experiences that include analysis of primary source documents are 
necessary elements of effective social studies teaching (Bailyn, 1994; Leinhardt, Stainton, & Virji, 1994; 
Wineburg & Wilson, 1991; Young & Leinhardt, 1998). This study examines the abilities of 70 fifth and 
seventh grade students to complete individual and multiple primary source document analyses based on their 
current background knowledge. Three researchers coded by grade, level of response (content), and type of 
processing (analysis of written discourse connectors and organization of written discourse) the completed 
document analysis tool responses. Rather than relying on others to interpret history for them, findings indicate 
that engaging students with primary source documents exercises the critical-thinking skills needed to analyze 
and interpret historical documents. However, this can be a very challenging task for students at this level. After 
connecting the study’s results to current research, the article concludes with a set of practical suggestions. 

Background 

Historians and history teachers describe history as an interpretive, constructive, analytic, and dialogic process 
— a discipline concerned with both knowledge of the past and the acts of constructing that knowledge (Bailyn, 
1994; Leinhardt, Stainton, & Virji, 1994; Wineburg & Wilson, 1991; Young & Leinhardt, 1998). Even with 
this knowledge, many students experience history, as currently taught, to be a passively learned subject that 
consists of memorizing dates and facts (Cantu & Warren, 2003; Stahl, Hynd, Britton, & McNish, 1996). Often, 
this pedagogical approach does not contribute to students’ historical understanding or positive attitudes toward 
the content. One way to create meaningful history learning experiences for students is to incorporate primary 
source analysis into classroom practice. 

Working with primary source documents is not easy for students or teachers. While there has been a 
nationwide and statewide call for students in our schools to be able to critically read, analyze, and interpret 
primary source documents (National Council for the Social Studies, 1994; New York State Education 
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Department, 1996), many teachers are not prepared to teach students how to succeed in meeting these goals. 
Engaging students in document analysis requires the use of complex skills, attitudes, and habits of mind 
(e.g., reading and comprehending, discovering main ideas, understanding the importance of perspective, 
synthesizing, revising hypotheses, writing clearly, presenting ideas orally, even working collaboratively) that 
need to be built over time (Kobrin, 1995). Clearly, the challenge lies in teaching these complexities without 
overwhelming students. Teachers are seeking ways to teach students more effectively at all levels to analyze 
primary source documents and ways to understand how to incorporate the use of primary source documents 
into their curriculum (Dobyns, 1992; Dutt-Doner, Cook-Cottone, Allen, & Rech-Rockwell, 2002; Kobrin; 
Price, 1990). 


Purpose of the Study 

The purpose of this research was to have a better understanding of the nature and development of the skills 
that aid students in analyzing primary source documents, the impact of students’ background knowledge on 
their analysis of primary source documents, and the process by which students create historical understanding. 
To fill the gap in empirical research and better understand the critical middle school years, this study focused 
on students in the fifth and seventh grades. 

Improving Guided Practices 

Primary source documents have long been used to enhance student learning in high school and college courses 
(e.g., Grant, Gradwell, & Cimbricz, 2004; Kobrin, 1995). In comparison to the growing accumulation of data 
demonstrating the effects of using primary sources at the high school level, however, studies on the use of 
primary source documents at the elementary and middle levels have been more limited. 

Review of the elementary and middle school-based literature reveals that a better understanding of the 
developmental skills and teaching involved in utilizing primary source documents for the purpose of having 
students along the developmental continuum respond to document-based questions is needed. Students need 
to be equipped with the tools to make sense of historical documents, and teachers need to know how to aid 
students in this process (Lee & Ashby, 2000; Ravitch, 1990). Data from this study will enable educators 
and teacher educators to understand more clearly the complex nature of document-based analysis to improve 
instructional practice in classrooms. 

Research Questions 

This research project was aimed at answering the following research questions: (1) What appear to be the 
essential skills involved in document analysis? (2) What are the skills that best predict successful document- 
based question response outcomes? (3) Are there significant differences between fifth and seventh grade 
students in these skills? 


Review of the Related Research 


Primary Source Document Use 

Social studies teachers, historians, and teacher educators identify several purposes for using primary sources 
in the classroom. First, as indicated previously, primary source analysis appears to promote a higher level of 
critical thinking and improved comprehension (Callison & Saunders-Brunner, 2004; Miller, 1998; Morgan, 
2002). However, this is only part of the primary source document pedagogical utility. In an effort to combat 
Cantu and Warren’s (2003) apparent student paradigm that “social studies is boring,” primary source 
document analysis offers students an opportunity to connect to history in a more personal, human, and active 
manner (Lafaye, 2001; Levstik, 1989, 1993; Maynard, 2003; Otten, 1998). That is, the student becomes 
the historian, the person who seeks out and discovers the truth. Now, for the student, it becomes a journey 
of personal discovery as opposed to the story of someone else’s journey of discovery. Therefore, learning 
becomes moving and alive. 
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Rather than acting as passive learners of history, students can act as historians or researchers examining 
primary source documents (Brophy, 1992) by concurrently developing historical understanding, curiosity, and 
a motivation to learn (Cantu & Warren, 2003; Morris, 2002; Otten, 1998; Winks, 1968). It is a very powerful 
process for students who are learning about the role that interpretation plays in historical understanding 
(Bruner, 1966). Specifically, researchers have indicated that students who use primary source documents 
have a fuller and more thorough understanding of history (Lafaye, 2001; Miller, 1998) and have a deeper 
understanding of how primary source documents can contradict one another due to issues such as author bias 
(Shiroma, 2000; Winks). In this way, primary source documents provide students with an opportunity to view 
multiple perspectives rather than a summarized version of history (Lafaye). 

While there appears to be a great deal of literature that provides “how to” information for teachers considering 
the integration of primary source documents, very little empirical research exists in this area. Teachers are 
making efforts to improve their teaching practices and integrate primary source documents as called for by 
state and national standards (e.g., National Council for the Social Studies, 1994; New York State Education 
Department, 1996), but they need to be assured by empirical findings that this practice is beneficial. As we 
have learned from research in other areas (e.g., Childhood and Adolescent Psychology), what might be best 
practice for high school populations may not be best practice for elementary and middle school populations 
(Cook-Cottone, 2004). 

Empirical Support for Classroom Practices 

Little empirical or qualitative research exists in the area of using primary source documents to teach social 
studies and the humanities that can guide teachers’ practices. Furthermore, most of this research has been 
done with students in high school, advanced placement, and college levels (Kobrin, 1995; Price, 1990; Stahl 
et al., 1996; Young & Leinhardt, 1998). Because of the nationwide movement requiring students as early 
as fifth grade to demonstrate mastery in responding to document-based questions, it becomes critical that 
educators identify the skills needed for middle school students to prepare for such a challenging standard. 

The research indicates that students must have a structured and nurtured development of the skills necessary 
in order to analyze documents (Lee & Ashby, 2000; Spoeher & Spoeher, 1994). This requires that educators 
understand what skills are critical to students’ success in analyzing documents, how to effectively scaffold 
students’ analyses, and how students’ developmental capacities affect teaching these skills (Brush, Saye, & 
PIHNet Development Team, 2005) 

High school level findings. Much of the research on primary source documents with high school students 
focused on identifying how students analyze primary source documents when given a document-based 
question and can yield some application for middle school teaching. To illustrate, Wineburg (1991) compared 
the think-aloud analysis of primary documents by novices (i.e., high school seniors) and experts (i.e., 
university historians) to better understand how each group reasoned about historical evidence. He suggested 
that differences between the two groups have less to do with historical knowledge than they do with an 
understanding of how historical knowledge is constructed. His findings indicated that early and frequent 
exposure to primary source documents might better prepare students to learn how to develop their own 
historical interpretations and negotiate the complex nature of others’ historical interpretations. 

Young and Leinhardt (1998) studied the development of five Advanced Placement History students’ written 
responses to four document-based questions over a year. They found that students had difficulty managing the 
complex layers of answering document-based questions. However, as students gained experience, they moved 
from knowledge telling (i.e., listing period and document content as discrete information bits) to knowledge 
transformation (i.e., integrating content as interpreted evidence for an argument). Their findings indicated that 
exemplary social studies instruction should include using a wide array of primary and secondary sources to 
provide students with opportunities to construct historical stories rather than to memorize facts. 

Finally, Stahl et al. (1996) studied 19 tenth grade students in an Advanced Placement US History class. 
Participants analyzed five or six documents about the Tonkin Gulf incident and summarized the documents to 
form an opinion. They found that the students addressed the problems using a sequence of behaviors, which 
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include (a) selection of ideas in each text read, (b) processing of ideas within that text, (c) creation of a mental 
model of the information, and (d) integrating ideas across texts to produce a final product. This model can be 
used by teachers at all levels to break down the process of analyzing primary source documents into smaller, 
more manageable skills for students as they are developing their capacity to handle the complex nature of 
document analysis. 

Other literature is less empirically focused, but provides practical suggestions for teachers as they incorporate 
primary source documents into their practice. For example, Musbach (2001) identified five rules that teachers 
should follow when using primary sources. Ferguson (1986) recommended a set of guidelines for the use of 
primary documents in heterogeneous classrooms. Moreover, Kobrin (1995) drew four conclusions that serve as 
guiding principles for teaching practice when incorporating primary documents. These practical suggestions 
often included discussion of working collaboratively, developing student skills, selecting documents, 
developing students’ background knowledge, providing document analysis tools, and developing the purpose 
for examining primary documents. 

While these research studies are helpful to educators in better understanding the complexity of document- 
based analysis and some of the practical implications of using primary source documents, it is important 
to note that the primary participants were above-average high school students. Findings from these studies 
support the use of primary source documents in the classroom (Stahl et al., 1996; Wineburg, 1991) but identify 
the challenge that teachers face in developing the necessary skills and “habits of mind” to actively engage 
students in the analysis process. In order to engage critical thinking skills at this level, the research indicates 
that frequent and early exposure to primary source analysis is necessary (Young & Leinhardt, 1998). In 
addition, findings from research indicate that it is just as critical to develop students’ attitudes and experiences 
with primary documents so that they approach document analysis in the same way as historians do (Wineburg). 
An examination of the research conducted with middle school students offers even more research-based 
knowledge. 

Middle school level findings. Lee and Ashby (2000) reported that between the ages of 7 and 14 there is a 
“broad shift with age from seeing history as stories ready-made and simply retold, to stories told by historians 
who find, compile, and collate information, to stories told by historians who actively produce their stories” (p. 
209). This was consistent with the shift in skills inherent in the age-expected developmental movement from 
concrete to formal operational thought (Davies, 2004). Lee and Ashby found that it was not until sixth grade 
(typically age 11) that some students (10%) were able to understand differences in historical accounts are 
due to problems with sources (e.g., mistakes, transmission errors, inaccuracies, biases, lies) or interpretation. 
The middle school years may be the platform for this developmental shift in ability or a time of coming of 
age for independent historical reasoning with primary source documents. To illustrate, Foster and Yeager 
(1999) studied fifty-one 12 year olds’ written and oral responses to conflicting written and pictorial historical 
sources. Foster and Yeager concluded that 12 year olds are capable of abstract historical reasoning, capable of 
a “process of constructing, reconstructing, and interpreting past events” (p. 286). 

Other research at the middle school level conflicts with the findings of Lee and Ashby (2000) and Foster and 
Yeager (1999). For example, Harris (2002) studied 53 gifted eighth grade students’ use of online primary 
source documents as part of a history unit. Findings indicate that students analyzed primary sources from 
the vantage point of their personal experience and current time frame; and that without a more structured 
framework, their incomplete knowledge did not allow them to write credibly. As Afflerbach and VanSledright 
(2001) noted, “Embedded [primary] texts and sources may contribute to students’ immediate reading 
experience and to their developing understanding and appreciation of history. Yet, without teacher assistance 
these texts are extraordinarily demanding of some students” (p. 704). 

The practical work of Massich and Munoz (1996) and Harris (2002) indicated that students must make 
personal connections to primary documents before any historical understanding can be achieved. The 
described classroom experiences showed that working with primary documents can provide opportunities to 
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make those connections and support the development of reading and writing skills while fostering an empathy 
and appreciation for the generations of those that came before them (Harris; Massich & Munoz). 

While there has been some research at the middle school level and students’ ability to work with primary 
sources, there is certainly not the breadth of research found for the high school level. Given the developmental 
paradigm of this age, which includes some students moving from concrete to abstract thinking in fifth grade 
and others not mastering these types of cognitive skills until much later, teachers need to know much more 
about the expectations for this group from both a developmental and historical perspective. They also need 
some practical ideas of what types of scaffolding could work with their middle school students as they learn 
and grow. 

Toward an Empirical Understanding of the Development of Document Analysis Skills 

Overall, there appears to be an emerging understanding of the skills involved in the analysis of documents, 
the context and implications for successful document analysis, and a growing body of practical suggestions 
for teachers. Accordingly, this research study was designed to add to the understanding of the nature and 
development of the student skills at the middle school level that aid in analyzing primary source documents, 
the impact of students’ background knowledge on their analysis of primary source documents, and the process 
by which students create historical understanding. To fill the gap in empirical research and better understand 
the critical middle school years, this study focused on students in the fifth and seventh grades. Further, as this 
particular study focused on student skills and development, no historical teaching was provided. That is, the 
students were given instructions and comments as if they were taking a document-based question exam. The 
specific research questions include: (1) What appear to be the essential skills involved in document analysis? 
(2) What are the skills that best predict successful document-based question response outcomes? (3) Are there 
significant differences between fifth and seventh grade students in these skills? 

Methods of Inquiry 

Data Sources 

Participants for this study were 18 private school students from 2 fifth grade classrooms and 52 private school 
students from 3 seventh grade classrooms. The private school is a 5-12 independent day school with about 
549 students in the school and class size averaging 12 students. Students took an entrance exam as part of the 
admissions selection process. Because of the school’s curriculum and independent status, it did not follow 
New York State regents’ standards. Students tended to be intelligent, highly motivated, and good readers. 

For this study, the researchers included students with parental consent in the fifth and seventh grades of the 
volunteering school. Each of the groups of students had previous experiences in analyzing primary sources 
but in most cases, the teacher scaffolded these experiences and small group assignments. Analyzing primary 
source documents independently was a new experience for almost all these students. Participants analyzed 
individual primary sources using analysis tools first on the related topic of the San Francisco earthquake of 
1906 and then collectively analyzed the documents using an analysis tool. The task was structured to replicate 
the structure of the document-based analysis they experienced on the fifth grade statewide exam. Participants 
had not previously studied the San Francisco earthquake of 1906, though they had learned about earthquakes 
in science and had prior knowledge about the causes and impact of an earthquake. 

The fifth grade classrooms have 12 desks each that form a horseshoe facing the whiteboard. Students sit in 
assigned seats, given the size of the two classes used in this study, there were empty seats scattered around 
the horseshoe. The seventh grade classrooms have a large room and again the seating is in a horseshoe shape 
facing the whiteboard and the teacher’s desk is in front of the board. Students do not have assigned seats so 
their seating varied during the multiple day exercise. Seating was not changed in either the fifth or seventh 
grade classroom for the primary source exercise. 

Data Collection Procedures and Tools 

A multi-layered data collection approach was used to answer the research questions stated so that multiple 
data sources could triangulate findings. The study was conducted over a two-week period in each of the five 
classrooms during times of the day when social studies would normally be taught. To ensure consistency in 


© 2007 National Middle School Association 


5 



RMLE Online — Volume 30, No. 6 


directions provided to students, the school library media specialist (SLMS) conducted the processes of the 
study in each classroom. She also had a regular presence in supporting the teaching that took place in the 
classroom. After being introduced by the classroom teacher, the SLMS told the students that they would be 
looking at some documents about the San Francisco earthquake of 1906 and would be asked to “analyze” the 
documents. She provided an explanation of the word “analyze” as needed. The SLMS told the students that 
she wanted to know everything they knew about earthquakes, the 1906 earthquake, 1906, or San Francisco. In 
multiple visits, the SLMS presented the students with the documents to be analyzed and the analysis guide to 
be used. She explained what the content of the document (e.g., a photograph of Market Street in San Francisco 
after the earthquake) and asked the students to complete the analysis guide. When students were unsure how 
to answer questions, the SLMS told the students to write what they thought. If they could not come up with 
an answer, they were directed to write “do not know” as a response. The students only received help to clarify 
vocabulary or instructions. The SLMS documented anecdotal field notes during data collection in order to 
provide a snapshot of the classroom experience during document analysis. While the teacher and SLMS 
remained in the room during the document analysis, they did not provide any scaffolding for students, as this 
research was focused on understanding the developmental capacity of individual students to analyze primary 
source documents at different grade levels. 

Background knowledge. To gain some insight into individual students’ knowledge of the San Francisco 
earthquake, students were asked to write everything they knew about this historical time, event, and place. The 
background information tool required each student to complete a series of four open-ended response questions 
prior to viewing and analyzing the documents. These questions addressed students’ general background 
knowledge about earthquakes, San Francisco, and the time. In addition, specific background knowledge was 
queried regarding the San Francisco earthquake of 1906. 

Individual primary source document analysis. In each classroom, students individually viewed a set of four 
primary source documents regarding the San Francisco earthquake of 1906, including two pictorial and two 
written (one newspaper account and one journal account). The different perspectives of the same historical 
event represented by the four documents provide a gauge of the students’ ability to integrate sources and think 
historically. Every day the students were given one document, from the set of four, to analyze individually. 
Students utilized the appropriate document analysis tool from the National Archives specific to the type of 
primary source, which had been modified by the researchers. Document analysis tools from the National 
Archives were chosen because the tools required in-depth analysis and were developmentally appropriate for 
students across varying grade levels. The image analysis tool included 11 open-ended questions that evaluated 
students’ observation, critical thinking, and inquiry skills. The written analysis tool included 11 open-ended 
questions that evaluated students’ reading comprehension, and heavily emphasized critical thinking and 
inquiry skills. 

To introduce each document, the SLMS read the documents aloud to the students before having them complete 
the analysis tool and provided a brief overview of the image sources so that the students understood the 
context. Then, students completed the document analysis individually. 

Multiple primary source analysis. On the final day of the study, the SLMS asked students to use the multiple 
document analysis tool to evaluate and synthesize the four documents collectively as well as to reflect on the 
process of forming an opinion regarding the documents. The multiple document analysis tool was composed 
of nine higher order, open-ended questions that allowed students to synthesize their analyses through (a) 
recognition of different perspectives and voices, (b) judgment of credibility and bias of sources, (c) negotiation 
of historical truth, (d) development of evidence-based arguments, and (e) document-based historical inquiry. 
Students were given a brief review of the primary sources that replicated what they were told when they 
individually analyzed the documents. They were also provided with their individual primary source analysis 
responses to use as they analyzed the documents collectively. 
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Treatment of Data 

Definition of variables. Variable parameters were constructed through prior knowledge (i.e., theory and 
previous empirical findings) and current experience (i.e., the coding process of this data set). That is, variable 
parameters were defined, in part, by previous research findings of the cognitive processes identified as 
potentially contributing to successful document analysis, integration, and subsequent question response 
(e.g., Lee & Ashby, 2000; Stahl et al., 1996) and from the responses of the participants in the study, both 
expected and unexpected. Accordingly, variables were explicated as explicitly as possible. Using this method, 
the researchers identified six major variables to examine across the student responses on document analysis 
worksheets. 

Coding of data. The researchers developed a set of rubrics for the background information tool, each 
individual document analysis tool type (written and image), and the multiple document analysis tool (see 
Appendix A). Within the set of rubrics, each of the students’ written responses was given a numeric code 
reflecting the degree to which a particular variable was present in each response. The three researchers tested 
the rubrics for validity and reliability using random data sets. In order to ensure inter-rater reliability in data 
analysis, each researcher analyzed student responses on the document analysis worksheets using the rubrics. 
Student final item scores were calculated by averaging rater scores for each item. 

The Background Knowledge Variable was the ratio of the score on the background rubric to the maximum 
score of 16. A student who received the maximum score on each of the four facets of the rubric (e.g. 
background knowledge of earthquakes score = 1- 4, time period score = 1-4, San Francisco score = 1-4, 
and historical event score = 1-4) would then have a Background Knowledge ratio score of student score/ 
possible score 16. To illustrate the process, a level one response was a statement of incorrect response, general 
rewording of the question, or no response. A level two response was a statement that demonstrated partial 
knowledge of the background area. A level three response was a statement that indicated general correct 
knowledge of the background areas. A level four response on any of the background facets indicates specific 
and correct knowledge of the particular aspect of background knowledge. Background knowledge responses 
were perhaps the most basic to analyze. 

The same type of formula was used for each variable; however, the processing and skill application were a bit 
more complex. For example, the Image Skills Variable assessed level of observation linked to artifact, detail 
of observations, and connection of observations to historical event, and includes student performance on both 
pictorial documents. As with the other skills, the student’s ability to observe, interpret, and record thoughts 
were all necessary for a successful response. For example, the first image analysis score was based on the 
level of observation linked to presented artifact. A level one score suggested that the student wrote incorrect 
responses or failed to respond. A level two score was given when student responses were concrete, heavily 
anchored on the questions asked, and if there was one response per question. A level three score was given 
for concrete description of image and any evidence of interaction (e.g., mood, emotion). Finally, a level four 
response was given if there was strong evidence of interpretation (e.g., connections among objects in photos, 
strong or several interpretations about emotions or mood). The Written Skills Variable assessed the use of 
information from the documents and understanding of author bias, and included student performance on both 
written documents. The assessment structure paralleled that used with the image skills variable. 

The Background Use Variable measured integration of prior knowledge in response; that is, this variable 
measured the use of background knowledge to respond to analysis questions across all documents, both written 
and image. To illustrate this scoring system, a level one response would have included only responses to the 
image or written document and failed to integrate any background knowledge about San Francisco, 1906, or 
the earthquake. A level two response included a little (i.e., one integration), a level three showed some (i.e., 
two to three integrations), and a level four showed strong integration (i.e., four to six integrations). Similarly, 
the Historical Applications Variable measured evidence of historical understanding in the response, across 
each type of document. Within the parameters of this variable, responses were assessed for evidence of an 
understanding of history as the process of constructing, reconstructing, and interpreting past events and that at 
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each processing of the event, the photographer, journalist, historian, and student each add a layer of subjective 
interpretation (i.e., 1 = none, 2 = little, 3 = some, 4 = strong). 

The Total Document Skills Composite was an aggregate variable of the student’s total performance on each 
document in addition to his or her total performance on the multiple documents set. The multiple document 
rubric set included scores in each of the following areas: (a) integration of documents, (b) understanding 
of multiple points of view, (c) negotiation of multiple documents to find “truth,” (d) authors’ agenda and 
critical review of document bias, and (e) evidence of historical understanding. Integration of document score 
was higher the better able the student was to integrate documents in his or her response. The understanding 
of multiple points of view score increases as the student responses demonstrated an understanding that 
documents represented potentially different points of view. To receive a high score on the negotiation of 
multiple documents score, the students was required to demonstrate an understanding of author and historical 
context influence on document information. Finally, student responses were assessed for evidence of a critical 
review of document bias. 

Statistical treatment. The researchers analyzed data in several ways. First, frequencies, means, and standard 
deviations were calculated for items and variables. Then, items and constructs were correlated and correlations 
were compared. A stepwise multiple regression/correlation analysis was conducted. In this analysis, an 
available list of potential explanatory variables was searched repeatedly for variables that could be included in 
the model. The best explanatory variable was used first, then the second best, and so on. Finally, an Analysis of 
Variance (ANOVA) was used to compare fifth and seventh graders on the key constructs. 

Analysis of qualitative data. Field notes from the researcher working in each classroom were recorded during 
the data collection in order to provide greater perspective to empirical data. These field notes were analyzed 
for themes that served to substantiate or complement the statistical results. 

Results 

Overall, 70 students completed the multi-level document analysis procedure. Of those, 25% (n = 18) were 
in the fifth grade and 75% (n = 52) were in the seventh grade. In general, both the fifth and seventh grade 
students demonstrated basic-level processing of historical documents (see Table 1). In each case described 
below, students’ highest achievable score was 1.0, and the highest achievable average rubric score (ars) of 4.0. 

On the Background Knowledge Tool (BG), the fifth grade students demonstrated a mean of .34, and a standard 
deviation of .10. Seventh grade students showed a mean of .43 (ars = 1.36), standard deviation of .19 (ars 
= 1.72). In reading student responses to the background knowledge worksheet, it was apparent that many of 
the students had little prior knowledge that could support them during the analysis of the documents. Their 
background knowledge was strong in the area of earthquakes as evidenced by responses like the following 
illustrative response: 

Earthquakes are the result of tectonic plates moving under the earth’s surface — often have loud noises or 
moving of the earth when are in action — they are rated on the Richter scale. In the oceans, they can create 
tsunamis. 

But, their background knowledge about the time period (1906) and the San Francisco earthquake were much 
more limited as evidenced by the large number of students who did not respond to these questions at all. 
However, even their misconceptions were evident as illustrated in the following responses that received the 
lowest rating on the rubric when asked what they knew about the time period: 

There was a gold rush. 

It was before Christopher Columbus. 

It was just after WWI. 
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When asked what they might know about the San Francisco Earthquake of 1906, the following responses 
illustrate their specific background knowledge that received the lowest rating on the rubric: 

It was in 1906, in San Francisco — lots of people died. 

It was an earthquake in San Francisco, California. 

The researchers expected these responses about the San Francisco earthquake of 1906. The researchers knew 
that the students had not studied this event and had not anticipated a great deal of background knowledge. The 
tool provided the researchers with an opportunity to see how the primary source documents about the San 
Francisco earthquake of 1906 might aid in the students’ knowledge about the historical event. 

In the area measured by the construct Image Skills (IS), the fifth and seventh grade students manifested means 
of .41 (ars = 1.64) and .51 (ars = 2.04) and standard deviations of .11 and .11 respectively. For this particular 
variable, seventh grade students surpassed their fifth grade counterparts in level of detail in response and 
connection of observations linked to artifact and historical event. During the analysis of the images, the 
researcher noted that “the fifth graders were disgusted that the photos were not clearer and better — they had 
little concept of the age of the photo or the equipment that took it.” Seventh grade observations were more 
concrete and anchored to the artifact with higher levels of accuracy than fifth graders. As evidenced by the 
data, neither fifth nor seventh grade students’ averaged scores beyond the mid-range on the rubric. 

Written Skills (WS) were similar with means of .44 (SD = .12; ars = 1.76) and .50 (SD = .09; ars = 2.0) for 
the fifth and seventh graders. Understanding author bias is a complex process and students in this study 
demonstrated difficulty with using information in the document to consider how the author’s bias may 
influence their depiction of the earthquake. Furthermore, the researcher noted in observations of the analysis 
that “There was little recognition on the part of the fifth graders that the excerpt written by Jack Fondon was 
not part of a book or story, but his actual observations of the scenes.” In this case, students believed what they 
read was true and indicated little concern for the author’s perspective and its influence. 

Background Use (BU) was found to be at a mean of .34 (SD = .08; ars = 1.36) for fifth graders and at a mean 
of .40 (SD = .08; ars = 1.6) for seventh graders. In reviewing students’ responses to the document analysis 
worksheets, researchers evaluated their ability to apply prior knowledge to the artifacts in responding to the 
analysis question. Students were asked to explain why a photo might have been taken or a newspaper article 
written; when the artifact may have been created, who created the artifact, and what problems are illustrated 
in the artifact. In many cases, the low level of background knowledge indicated by findings described 
above became more apparent as demonstrating an inability to make the connection with basic knowledge 
to more complex understandings about the San Francisco earthquake of 1906. Fack of knowledge about the 
time period inhibited students from moving beyond basic observations of what they saw to more complex 
connections to people’s experiences while living through the earthquake. 

Historical Applications (HA) were similar at a mean of .33 (SD = .07; ars = 1.34) for fifth graders and a mean 
of .36 (SD = .06; ars = 1.44) for seventh graders. These fifth and seventh graders demonstrated little historical 
understanding about the San Francisco earthquake. In many cases, students left responses blank or wrote 
simplified responses. Finally, the Total Document Skills Composite (TF) had a mean score of 1.85 (SD = .35; 
ars = 1.23) for fifth graders and a mean of 2.22 (SD = .34; ars = 1.48) for seventh graders. 
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Table 1 

Fifth and Seventh Grade Means, Standard Deviations, and Ranges on Key Variables 


Var. 

Grade 

N 

Mean 

Std. Deviation 

Minimum 

Maximum 

BG 

5th 

18 

.34 

.10 

.25 

.63 


7th 

52 

.43 

.10 

.25 

.69 


Total 

70 

.41 

.11 

.25 

.69 

IS 

5th 

18 

.41 

.11 

.25 

.60 


7th 

52 

.51 

.11 

.28 

.81 


Total 

70 

.48 

.12 

.25 

.81 

WS 

5th 

18 

.44 

.12 

.25 

.65 


7th 

52 

.50 

.09 

.35 

.73 


Total 

70 

.48 

.10 

.25 

.73 

BU 

5th 

18 

.34 

.08 

.25 

.48 


7th 

52 

.40 

.08 

.27 

.63 


Total 

70 

.39 

.08 

.25 

.63 

HA 

5th 

18 

.33 

.07 

.25 

.48 


7th 

52 

.36 

.06 

.27 

.58 


Total 

70 

.36 

.07 

.25 

.58 

TL 

5th 

18 

1.85 

.35 

1.25 

2.38 


7th 

52 

2.22 

.34 

1.60 

3.13 


Total 

70 

2.12 

.38 

1.25 

3.13 


BG: Background Knowledge IS: Image Skills WS: Written Skills 

BU: Background Use HA: Historical Applications TL: Total Document Skills Composite 


Research question one. What appear to be the essential skills involved in document analysis? According to 
correlation findings, Background Knowledge, Image Skills, Written Skills, Background Use, and Historical 
Application are significantly correlated with each other as well as with the Total Document Skills Composite 
(see Table 2). In addition, the basic document skills (e.g., knowledge level comprehension skills) for images 
and written documents appear to be quantitatively different from the more complex interpretive skills 
(Background Use and Historical Applications). These construct differences are supported by the strong and 
significant correlation among the document skills items, and the less strong and, more often insignificant 
correlation with the use and application items. This suggests that these items are measuring unique 
constructs — basic skills versus complex skills (see Table 3). 
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Table 2 

Pearson Correlation among Background Knowledge, Document Related Subskills, and Total Score Performance 


Scale BG 

IS 

WS 

BU 

HA 

TL 

BG 1.00 

.51 ** 

44 ** 

40 ** 

32 ** 

.57 ** 

IS 

1.00 

.56 ** 

42 ** 

.35 ** 

.81 ** 

ws 


1.00 

.58 ** 

.51 ** 

.83 ** 

BU 



1.00 

7g ** 

.76 ** 

HA 




1.00 

70 ** 

TL 





1.00 

** Correlation is significant at the 0.01 level (2-tailed). 




BG: Background Knowledge 
BU: Background Use 

IS: Image Skills WS: 

HA: Historical Applications TL: 

Written Skills 

Total Document Skills Composite 

Table 3 

Sample Subskill Item Pearson Correlations (2-tailed) from Image One 



Artifact Links 
(Image Skill) 

Artifact Links Detail Quality 
(Image Skill) (Image Skill) 

1.00 .69** 

Image to Event 
(Image Skill) 

.82** 

Background Use 
.28** 

Historical 

Applications 

.24* 

Detail Quality 
(Image Skill) 


1.00 

.76** 

40** 

.26* 

Image to Event 
(Image Skill) 



1.00 

.21 

.19 

Background Use 




1.00 

04** 

Historical 

Applications 





1.00 


** Correlation is significant at the 0.01 level (2-tailed). 
* Correlation is significant at the 0.05 level (2-tailed). 


Research question two. What are the skills that best predict successful document-based question response 
outcomes? According to the findings indicated in Table 4, all of the variables entered in the stepwise regression 
analysis played a significant role ( p < .001) in the prediction of the total document skills composite. The 
construct Written Skills variable, which is composed of how a student used information from the document 
and his/her understanding of an author’s bias was the best predictor of the Total Composite, accounting 
for nearly all of variance (.98). However, the Image Skills variable, which includes the student’s level of 
observation as it is linked to artifact, detail of observations, and connection of observations to historical 
event demonstrated a small but significant contribution to the prediction of the total composite. In addition, 
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the Historical Applications variable demonstrating evidence of historical understanding in the response 
contributed to the prediction of the total skill composite, accounting for variance beyond what was explained 
by the previously entered variables. Finally, the Background Use variable that measured a student’s integration 
of prior knowledge in a response, while a significant predictor, had the least predictive value. 


Table 4 

Stepwise Regression Analysis: The Prediction of a Total Document Skill Composite 


Model 

R 

R Square(a) 

Adjusted 
R Square 

Change Statistics 








R Square 
Change 

F Change 

dfl 

df2 

Sig.F 

Change 

1 

.993(b) 

.987 

.986 

.987 

5092.659 

1 

69 

.000 

2 

.997(c) 

.995 

.995 

.008 

103.388 

1 

68 

.000 

3 

.999(d) 

.998 

.998 

.004 

145.816 

1 

67 

.000 

4 

.999(e) 

.999 

.999 

.000 

26.406 

1 

66 

.000 


a For regression through the origin (the no-intercept model), R Square measures the proportion of the 
variability in the dependent variable about the origin explained by regression. This cannot be compared 
to R Square for models, which include an intercept, 
b Predictors: WS 
c Predictors: WS, IS 
d Predictors: WS, IS, HA 
e Predictors: WS, IS, HA, BU 

BG: Background Knowledge IS: Image Skills WS: Written Skills 

BU: Background Use HA: Historical Applications TL: Total Document Skills Composite 


Research question three. Are there significant differences between fifth and seventh grade students in these 
skills? Findings of the ANOVA that compared the means of the fifth and seventh grade students on each 
of the key variables (See Table 5) indicated significant difference on nearly all constructs. Seventh grade 
students were significantly stronger than fifth grade students (p < .05) on the following variables: Background 
Knowledge, Image Skills, Background Use, and Total Document Skills Composite. The difference between 
the two groups on the variable Written Skills was approaching significance (p = .053). Interestingly, in the 
area of Historical Applications, there were not statistically significant differences between the fifth and 
seventh graders ( p = .098), which indicated that neither group was able to use the documents to achieve greater 
historical understanding. Observations by the researcher during the document analysis indicated that the fifth 
graders experienced much more frustration and they seemed unwilling to try when faced with challenging 
questions. The fifth graders “had little understanding of the term analyze.” In addition, the seventh graders 
appeared to “just want to get it done,” as observed by the researcher. 


© 2007 National Middle School Association 


12 



RMLE Online — Volume 30, No. 6 


Table 5 

Analysis of Variance : Document Subskills and Total Skills Scores ( Fifth and Seventh Graders) 




Sum of Squares 

df 

Mean Square 

F 

Significance 

BG 

Between Groups 

.115 

1 

.115 

10.915 

.002* 


Within Groups 

.717 

68 

.011 




Total 

.832 

69 




IS 

Between Groups 

.155 

1 

.155 

12.640 

.001* 


Within Groups 

.833 

68 

.012 




Total 

.987 

69 




WS 

Between Groups 

.039 

1 

.039 

3.875 

.053 (*) 


Within Groups 

.683 

68 

.010 




Total 

.722 

69 




BU 

Between Groups 

.043 

1 

.043 

7.133 

.009* 


Within Groups 

.407 

68 

.006 




Total 

.450 

69 




HA 

Between Groups 

.012 

1 

.012 

2.807 

.098 


Within Groups 

.286 

68 

.004 




Total 

.297 

69 




TL 

Between Groups 

1.757 

1 

1.757 

15.022 

.000** 


Within Groups 

7.954 

68 

.117 




Total 

9.711 

69 





(*) approaching, * P < .05, **p < .01 

BG: Background Knowledge IS: Image Skills WS: Written Skills 

BU: Background Use HA: Historical Applications TL: Total Document Skills Composite 


Discussion 

Data from this study indicate a number of important findings for teachers and teacher educators to consider, 
along with the practical suggestions that flow from the findings. In addition, it is important to frame the 
findings of this study within the context of current research so that educators can understand how the findings 
of this study compare to the existing research base. 

First, our findings suggest that background knowledge, document analysis skills for both written and images, 
the ability to integrate background knowledge, and historical thinking all contribute to the successful use of 
documents. The findings suggest that these skills collectively play an important role in successful document- 
based analysis and that teachers should pay careful attention to each one. For this sample of middle school 
students, written-document analysis skills were the best predictor of their Total Document Skills Composite 
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score. When students are able to extract key information from the documents, they are better able to use this 
information in their analysis. Students are more effective in their analysis when they are able to consider 
the role of the author’s bias in determining the validity of the document within the context of a group of 
documents. These findings are not surprising considering the findings of Stahl et al. (1996) and Massich and 
Munoz (1996), which have drawn attention to the role of language arts skills in primary source analysis. It is 
critical then for middle school teachers to understand the important role of developing language arts skills in 
supporting the analysis of primary source documents. Comprehending primary sources and extracting key 
information plays a critical role in document analysis. Students must have multiple experiences analyzing 
documents to assess an author’s point of view and how that should be considered. Developmentally, this is a 
challenging skill for middle school students and needs to be adequately developed. The curriculum should be 
mapped to integrate these subject areas so that the skill development is supported across content and not just in 
the social studies classroom. 

However, image analysis skills, historical understanding, and background knowledge integration all played a 
significant role in predicting a student’s Total Document Skills Composite score. These constructs all appear 
to play an essential role in a student’s ability to work with primary source documents. Image analysis skills 
include those in which students are able to carefully examine an image for details and connect that to the 
historical event and/or time period being studied. In this way, background knowledge integration becomes 
critical so that students can connect prior knowledge with new knowledge extracted from the document. Put 
in the context of a student’s historical understanding, these skills work interactively to aid in primary source 
analysis. For practicing teachers, these research findings indicate the importance of (a) activating students’ 
general and specific background knowledge to examine primary sources through direct teaching or research 
experiences (Kobrin, 1995), (b) developing knowledge level comprehension skills by reinforcing language arts 
skills, and (c) modeling and scaffolding connection of prior knowledge to primary source documents through 
teacher-led analysis. 

Second, it was clear from both the fifth and seventh grade analysis worksheets that primary source documents 
alone, as administered in this study, may not facilitate a deep understanding of the historical event. This was 
evident in Ihe average fifth and seventh grade scores clustering around rubric scores of 1 and 2, the lowest end 
of the scoring scale on the rubric. These scores reflect only basic observations, limited integration, and little 
if any historical understanding. Researcher observations during the analysis indicated, “All students seem to 
think there are ‘correct’ answers that they need to find and were concerned that answers were right.” These 
beliefs reveal that students do not understand the nature of historical understanding and how historians use 
primary source documents to interpret history. Clearly, this emphasizes the critical role the teacher and peer 
groups play in supporting document analysis. Similar to the findings of Lee and Ashby (2000) and Foster and 
Yeager (1999), both the fifth and seventh graders had difficulty managing the conflicting information from 
different sources, seemingly on the cusp of abstract historical reasoning. Teachers must provide opportunities 
for students to judge the relevance and accuracy of historical information so they are better able to determine 
a document’s credibility. Simply helping students understand that not all primary sources provide accurate 
and bias-free knowledge through repeated exposure and critical analysis is critical for this understanding. In 
addition, teachers can build “judging for relevance” skill by having students examine documents that provide 
conflicting information or only present part of the story to understand that examining multiple documents will 
best capture a sense of the historical event and/or time. Experiences with primary source documents need to 
be supported by teacher modeling and opportunities to collaborate with peers so that many perspectives can be 
presented and considered (Ferguson, 1986; Kobrin, 1995; Musbach, 2001). 

Third, our findings indicated that fifth and seventh graders were different in many areas, and there may 
be a developmental trajectory of historical thinking. The middle school years, uniquely, may be a critical 
period for the development of independent, historical thought (Foster & Yeager, 1999; Lee & Ashby, 2000). 
Significant differences between the two grades included levels and use of background knowledge, image 
analysis skills, and quite possibly, written document analysis skills. Qualitative review of the data suggested 
that the fifth graders rarely suggested other primary sources to find out additional information, but seventh 
graders did at times offer this possibility. Recommendations from Wineburg (1991) and Young and Leinhardt 
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(1998) are supported by our findings, which suggest that teachers need to provide consistent, successful use 
of primary source documents as a part of learning so students can see their importance in understanding 
the “real story.” In addition, it appeared that the fifth graders considered each primary source to be true, 
while the seventh graders were less likely to accept the credibility of primary source documents. With 
the support of developmental research, it is apparent that the middle school years are transformative in 
students understanding that history is actively produced through stories considering multiple perspectives 
(Davies, 2004; Lee & Ashby, 2000). Students need to have opportunities to examine authors’ motivations 
and perspectives, and to understand how personal viewpoints can color interpretation. The teacher must 
demonstrate the necessity of multiple points of view to determine the complete picture of the historical event. 

Finally, consistent with the findings of Spoeher and Spoeher (1994) the fifth graders had difficulty analyzing 
the primary source documents without significant prior preparation by the teacher. Because the focus on 
this study was on understanding the developmental nature of analysis skills, instruction did not precede 
the analysis presented to students in this study. Students were given an assessment to evaluate their prior 
knowledge so that we may better understand its impact on analysis. In addition to development of prior 
historical and content knowledge, the teacher needs to model the analysis process. At this developmental 
level, the teacher must provide experience in evaluating the document’s relevance, primarily by providing 
guided questions to scaffold analysis. The teacher must model the process of analysis and precede analysis 
with accurate background knowledge development. In this way, the background knowledge can support the 
document analysis instead of providing inaccuracies or misconceptions for knowledge development. 

Importance of study 

Findings from this study add to the limited research base on using primary source documents with students at 
the middle school level. From these findings, practical suggestions for teachers emerge that will better enable 
them to use primary source documents to enhance learning in their classrooms. In addition, the results of this 
pilot study will affect future studies on the use of primary source documents. Since relatively little research has 
been done in this area below the high school level, these data will provide a baseline for ways to conduct further 
research. The biggest issue surrounding research across developmental levels is how to capture what students 
are capable of at different grade levels using different, grade-appropriate primary source documents and 
analysis tools. Is it possible to identify the differences between the groups using different sets of materials? In 
addition, what can teachers do to help students below the fifth grade level to analyze primary source documents? 
In addition to these lines of related research, future studies might want to focus on the ways in which teaching 
methods can facilitate effective primary source analysis. While our study did not examine this relationship, 
findings indicate that further implications for classroom practice will result from this line of research. 
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Appendix A 


Background Information Rubric 


Analysis Type 

1 

2 

3 

4 

Level of 
Background 
Knowledge 

Earthquakes 

General knowledge 
incorrect 
No response 
Rewording 
of question 

Partial correct 

knowledge 

demonstrated 

General correct 
knowledge 

Specific correct 
knowledge 

Level of 
Background 
Knowledge 

San Francisco 

General knowledge 
incorrect 
No response 
Rewording 
of question 

Partial correct 

knowledge 

demonstrated 

General correct 
knowledge 

Specific correct 
knowledge 

Level of 
Background 
Knowledge 

1906/Time Period 

General knowledge 
incorrect 
No response 
Rewording 
of question 

Partial correct 

knowledge 

demonstrated 

General correct 
knowledge 

Specific correct 
knowledge 

Level of 
Background 
Knowledge 

SF Earthquake 

General knowledge 
incorrect 
No response 
Rewording 
of question 

Partial correct 

knowledge 

demonstrated 

General correct 
knowledge 

Specific correct 
knowledge 


Written Document Rubric 


Analysis Type 

1 

2 

3 

4 

Use of Information 
Provided in a 
Written Document 

Response appears 
to be opinion based 
No information from 
document integrated 
Inaccurate 
information 

Shows little use of 
accurate document- 
based information 

Shows some use of 
accurate document- 
based information 

Shows strong use of 
accurate document- 
based information 

Understanding 
of Author’s 
Perspective (Bias) 

Shows no under- 
standing of author’s 
perspective or 
agenda 

Shows little under- 
standing of author’s 
perspective or 
agenda (at least 1) 

Shows some under- 
standing of 
author’s perspective 
or agenda (2-3 
responses) 

Shows strong 
understanding of 
author’s perspective 
or agenda 
(4-6 responses) 
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Image Analysis Rubric 


Analysis 

1 

2 

3 

4 

Level of 
observation 
linked to artifact 
characteristics 

General answers 
that may 
be incorrect 
Strongly anchored 
on question 

Concrete 

descriptors 

At least one per 

question 

Observation 

anchored on 

question 

Concrete descrip- 
tors and ANY 
evidence of 
interpretation (e.g., 
mood, emotion) 

Strong evidence 
of interpretation 
(e.g., connections 
among objects in 
photos, strong or 
several interpreta- 
tions about mood 
or emotion) 

Detail and quantity 
of observations 

Less than one per 
question with 
inaccuracies 

Concrete 
At least one per 
question with one 
or less inaccuracies 

Multiple accurate 
observations 
Detailed observa- 
tions 

Many accurate 
observations with 
elaborate detail 

Inference of 
interaction related 
specifically to the 
depicted event 
in question (e.g., 
San Francisco 
earthquake) 

Concrete response 
General 

No connection to 
the specific event 

Concrete response 
General 

Little connection to 
the specific evenf 

Shows signs of 
inference or inter- 
pretation regarding 
the specific event 

Solid connection 
with event and 
strong effort to 
connect to specific 
event 
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Multiple Document Analysis Rubric 


Analysis Type 

1 

2 

3 

4 

Integration of 
documents 

No evidence of 
integration 
Draws information 
from background or 
one document 

Draws information 
from background 
and two documents 

Draws information 
from background 
and more than two 
documents 

Draws information 
from background 
and all documents 

Understanding 
multiple points 
of view 

Does not address 
the issue of multiple 
points of view 

Little inference to 
the concept of 
multiple points of 
view (1-2 responses) 

Some inference to 
the concept of 
multiple points of 
view (3^1 responses) 

Strong indication of 
the understanding 
that documents 
represent different 
points of view 
(5+ responses) 

Negotiation 
of multiple 
documents to 
find “truth” 

No evidence of 
integration or 
attempt to bring 
answer from 
documents 

Answers questions 
with a choppy list of 
different document 
findings 

Uses an 

averaging heuristic 
and a general 
summation process 
to answer questions 

Evidence of 
contextual and 
author’s influences in 
information and 
response is given 
within this context 

Author’s agenda 
and critical review 
of document (Bias) 

Shows no evidence 
of considering 
author’s agenda and 
critical review of 
documents’ 
representativeness 

Shows little 
evidence of 
considering author’s 
agenda and critical 
review of documents’ 
representativeness 

Shows some 
evidence of 
considering author’s 
agenda and critical 
review of documents’ 
repre sentati vene s s 

Shows strong 
evidence of 
considering author’s 
agenda and critical 
review of documents’ 
representativeness 

Evidence of 
historical 
understanding 

Responses show 
no evidence 
of historical 
understanding 

Responses show 
little evidence 
of historical 
understanding 
(1-2 responses) 

Responses show 
some evidence 
of historical 
understanding 
(3-4 responses) 

Responses show 
strong evidence 
of historical 
understanding 
(5+ responses) 
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